Text classification as measurement in social sciences

• Long tradition in social sciences to measure/operationalize concepts from text
• Content analysis : assign predefined categories to texts units to measure a social phenomena (Krippendorff 2018)
• Systematic measurement of (latent) concepts in texts, through coding rules and quantification
• Different from more qualitative/intepretative approaches of text analysis which focus on meaning, context, interpretation

Text classification through manual content analysis

• Historical use through manual content analysis
• Use of (multiple) Experts/RAs/crowdworkers to code texts
• Exemples :
  • Coding of news articles to measure media frames
  • Protest event analysis :eg. Poldem
  • Extensive datasets with high number of categories : Comparative manifesto project, Comparative Agendas projects

Advantages and limits of manual content analysis

• Importance of human judgment, context-knowledge

• High quality data, in comparison to automated methods such as dictionnaries

• Highly time consuming, human labor intensive and costly

  • Eg : Manifesto Project, CAP

• Often need to rely on a small sample of texts, which can be biased

  • Eg. Protest event Analysis

Let the machine come in

• Text classification can be automated with supervised machine learning
• One of the main use cases of text analysis in social sciences
• “Augment” the human, does not replace it
• Rather than coding all the texts manually, we can train a model to predict the categories based on a sample that we have coded
• Supervised : we need a labeled dataset to train the model
• Unsupervised : we do not need a labeled dataset, the model will learn the categories by itself
• The model will learn the text features that are associated with the categories

Some common machine learning terms explained

Inputs and outpus in supervised learning for text analysis

• Input : text features + labels
• Model : a function that will learn the relationship between the text features and the labels
• Ouput :
  • Probabilities of belonging to each category
    
  • If two categories, a binary classification, usually > 0.5
  • If more than two categories, a multi-class classification, class assign is highest probability

Uses cases

• Policy issue classification (Burscher, Vliegenthart, and De Vreese 2015; Burst 2024)
• Populism, Nationalism, Authoritarianism (Licht et al. 2024; Bonikowski, Luo, and Stuhler 2022)
• Mentions of social groups (Licht and Sczepanksi 2024)
• “Off the record” statements in mainstream media (Do, Ollion, and Shen 2024)
• Nostalgia in party manifestos (Müller and Proksch 2024)
• Stance detection : eg. towards Donald Trump (Burnham 2024)
• In your daily life : spam detection, credit card fraud detection, content moderation on social media, sentiment analysis of Amazon reviews, language detection

Supervised text classification pipeline

1. Getting a clean corpus of labelled texts
2. Transforming texts into numerical features
3. Model training
4. Model evaluation
5. Inference on full corpus
6. Validation
7. Use in downstream analysis

Licht et al. (2024) : A supervised learning workflow

# Getting a clean corpus of labelled texts

Getting a clean corpus of labelled texts

• Training a “classifier” requires labelled data : texts with categories assigned
• Labels are used to :
  • Train the model
  • As a gold standard to evaluate the performance of the model
• Existing labels
  • Potential labels with similar categories from other projects
  • Labels in the wild : metadata from datasets/website
• Most of the time, we need to label the data ourselves

Which texts to annotate ?

• From a corpus of texts, we need to select a sample to annotate
• Series of decisions to make
• Text unitization : documents, paragraphs, sentences ?
• Sample size : how many texts to annotate ?
  • Usually the more the better
  • But depends on number of categories, complexity of the task
• Sampling strategy : random, stratified ?
  • Important to have a representative sample

How to annotate ?

• Coding rules, codebook, to be refined
• Quality of annotation is super important : garbage in, garbage out
• Has to be systematic, replicable
• Get to know the data

Who annotates ?

• Most of text classification projects rely on RAs or crowdworkers
  • Trained on the codebook and paid for their work
  • Quality of annotations checked with inter-coder reliability (ICR) tests
• As phd students, we often do it ourselves
  • Time consuming, but we know the data better
  • But if you want to publish, you need to show ICR
• More and more use of LLms to annotate texts : highly discussed in the literature

Transforming texts into numerical features

• Texts are not directly usable by machine learning algorithms
• Need to convert them into numerical features
• Different possibilities :
  • Bag-of-words (BOW)
  • Term frequency-inverse document frequency (TF-IDF)
  • Static word embeddings (Word2Vec, GloVe)
  • Contextual word embeddings (eg. BERT)

Supervised learning models

• Split the annotated sample into a training set and a test set (70/30 rule)
• Train a model on the training set that learns the relationship between the text features and the labels
• Return the probability of a text belonging to a category

Different types of models

• Classical models : Logistic regression, Naive Bayes, SVM, Random Forest
  • Simple and fast
  • Rely on bag-of-words representations
  • Lot of preprocessing needed
  • Limited performance
• Transformers models (eg: BERT)
  • Contextual representation of texts
  • No need for extensive preprocessing
  • Transfer-learning paradigm : models have already language knowledge
  • Computationally intensive

Evaluating performance of the model

• Use the test set to evaluate the performance of the model

  • Set of texts that the model has not seen but for which we know the categories
  • “Held-out” data/Gold standard

• The model is used to predict the categories of the texts in the test set

• The predictions are compared to the true categories with different metrics

• Accuracy : proportion of correctly classified texts (highly limited for imbalanced datasets)

\[ \ \text{Accuracy} = \frac{\text{Number of Correct Predictions}}{\text{Total Number of Predictions}} \ \]

Confusion matrix

• Positive class : the class we want to predict
• Negative class : the other class

Recall, Precision and f1-score

• Recall : proportion of actual positive cases that were correctly classified

\[ \ \text{Recall} = \frac{\text{True Positive}}{\text{True Positive} + \text{False Negative}} \ \]

• Precision : proportion of predicted positive cases that were correctly classified

\[ \ \text{Precision} = \frac{\text{True Positive}}{\text{True Positive} + \text{False Positive}} \ \]

• f1-score : harmonic mean of recall and precision

\[ \ \text{f1-score} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} \ \] ## Recall and precision trade-offs {.smaller}

• Recall and precision are often in trade-off
  • Increasing recall involves predicting more positive cases at the risk of increasing false positives (reduce precision)
  • Increasing precision involves predicting less positive cases at the risk of missing some positive cases (reduce recall)
• The model may be optimized to maximize one of the two, depends on the task
  • Eg. in a spam detection task, we want to maximize precision : we prefer to have some spam in our inbox rather than missing an important email
  • Content moderation/hate speech : we often want to maximize recall : we prefer to have some false positives rather than missing a hate speech

What is a good performance ?

• Maximization of one metric, usually f1-score, but depends on the task
• In general, we look for at least f1-score > 0.7
• A “good” performance depends on the complexity of the task
  • Conceptual complexity
  • Task complexity : multi-class, imbalanced classes
• But, sometimes we do not want our model to be too good neither : overfitting

How to improve performance

Measurement validity

• Face validity
• Convergence validity

What to do classification outputs ?

• It is all about measurement !
• Often used as to measure previously unmeasured concepts
• Often some aggregated measure of the categories is used
• Descriptives : variation over time, across countries, groups
• Downstream tasks :
  • Use measurement as DV in a regression
  • Use measurement as IV in a regression
• But careful because : measurement error can bias the results